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ABSTRACT 



A key objective of a federally funded grant received by the 
Wake County Public Schools (North Carolina) was to develop new methods of 
assessment. The school district decided to fulfill this objective by 
developing student electronic portfolios. In spite of careful planning, the 
development of the electronic portfolios was filled with difficulties. Staff 
members persevered, developing electronic portfolios geared to the programs 
of six magnet elementary, middle, and high schools. Evaluation of the 
portfolio development process resulted in the identification of five specific 
lessons: (1) it may be better to develop traditional portfolios first and 

then mediate them; (2) schools should start small with this type of 
technology; (3) schools should take the time to find the right software for 
the electronic portfolios; (4) both students and teachers need to have access 
to the portfolio at any time from any place in the school; and (5) technology 
support is essential. (Contains 4 tables and 13 references.) (SLD) 
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Introduction 

Changes in teaching and learning stemming from the education reform 
movements of the late 1 980s generated calls for new methods of assessment. The 
educational literature of the early to mid 1 990s was replete with espousals of 
alternative assessment. Books and articles enumerated the advantages of performance 
and portfolio assessments, and related staff development materials proliferated 
(Herman, Aschbacher, & Winters, 1992; Fogarty, 1996). Some states added 
performance assessments to their accountability systems, and national testing 
programs developed performance-based items. Many articles and professional 
development materials during this period described the benefits and efficacy of new 
assessment methods (Mitchell, 1 992). There was debate about the reliability and 
validity of new assessments (Messick, 1 995), but little discussion about the resources 
necessary to develop and implement them, particularly when technology was involved 
in the development process. 

One key objective of a federally funded grant received by the Wake County 
Public School System (WCPSS) in Raleigh, NC, was to develop new methods of 
assessment. Based on their research about and preview of assessments, teachers, 
administrators, and technology specialists at the six magnet schools in the grant 
decided to fulfill this objective by developing student electronic portfolios. Across the 
three years of grant implementation, achievement of this objective met with varying 
levels of success, and it is now possible to evaluate the benefits of these assessments; 
but, more importantly, to assess the difficulties in their development. 

While pioneering electronic portfolios, there were times when grant staff 
considered "circling the wagons." In spite of careful planning, their journey was 
fraught with difficulties and delays. Because of glitches in portfolio software and the 
late arrival of computer hardware, assessments scheduled for completion in the first 
year had to be piloted with samples of students. Initial delays affected the entire 
timeline, but rather than circling the wagons, staff members were creative in their 
adaptations. They continued the journey — at a slower pace but still in a positive 
direction. Instead of complete portfolios planned for the first year, some schools 
developed specific segments to be plugged into the whole at a later date. Other 
schools kept the planned assessments intact, but implemented them with much smaller 
groups of students. 



Methods 

Part of the grant evaluator's role was to oversee development of the alternative 
assessments. A mixed-method approach was used to evaluate this aspect of the grant 
(Greene & Caracelli, 1997). The evaluator attended portfolio planning meetings of the 
grant leadership team, keeping field notes and developing tables to depict and describe 
similarities and differences in the approaches used at each school (Tables 1, 2, & 3). 
She visited schools to observe during all stages of the development process and 
returned to view completed portfolios for a sample of students. After grant staff 
members attended national meetings and visited other schools and districts to gather 
information about electronic portfolios and portfolio software, they met with the 
evaluator to discuss their findings and how this information would affect the portfolio 
process at each school. 

Magnet schools in the grant offered a variety of themes ranging from Aits and 
Science or Montessori Programs at the elementary schools, Academically 
Gifted/Gifted and Talented Programs at the middle schools, and Interdisciplinaiy 
Teaming or Accelerated Learning at the high schools (Table 1). The specific objective 
that the portfolios addressed was "to explore and establish new methods for assessing 
student progress related to new or revised magnet themes" (WCPSS, 1995). Portfolios 
were tailored for the specific theme at each school, so their formats differed. The 
evaluator reviewed portfolio output in both electronic and print formats, and she 
interviewed administrators and magnet staff about the effectiveness of the development 
process and the utility of the portfolios. Focus groups were used to identify concerns 
of technology specialists working at grant schools. Other data sources included: field 
notes of magnet staff meetings; regular on-site observations at each school; tables of 
similarities and differences by school — revised and tracked over time; debriefings of 
magnet staff after trips to conferences and other districts; and audits of electronic and 
printed versions of portfolios. 

The Development Process 
Software Selection and Hardware Issues 

Having decided to "establish new methods for assessing student progress" 
by developing electronic portfolios, staff members needed to select software for that 
purpose. Through the grant, each school had a staff member one-third to one-half time 
who had technology training and certification. In the fall of 1995, they assisted other 
staff members in reviewing available software with capabilities for developing 
portfolios. There were numerous data-base "plus" types of programs for storing 
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Table 1. Schools & Magnet Themes, Year 1 
Software, Student Sample, Professional Development, and Type of Assessment 
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student information, photos, and class work, but few programs with capabilities for 
capturing and organizing audio, video, and graphics exhibits. From the few with these 
additional capabilities, three schools chose Grady Profile (Aurbach Associates), one 
decided to use Hyper Studio, and one selected a Hyper Studio-type package that would 
run on both Mac and IBM computers. The sixth school, a new high school being 
planned for 1997, later decided to use IBM School Vista. Even though portfolio 
software choices were similar, usage differed because of the varied magnet themes and 
grade levels of the schools (Table 1). 

The technology specialists reviewed computer hardware and related software 
programs available at each school to determine the amount of existing support for the 
portfolio software. According to the district-wide technology plan, local area networks 
(LANs) were scheduled to be in place at most grant schools by November 1995, and 
the LANs were factored into the schools' ability to support portfolios and store related 
files. The grant budget provided some funding for equipment to supplement what was 
available at the schools. As November 1995 came and went without installation of the 
LANs, it became clear that storage devices would be needed for the portfolios, which 
would contain data-dense audio and video files for large numbers of students. 

As with the LAN installations, other technology-related delays arose early on, 
e.g., late arrival of the portfolio software and/or computers on which to run it and 
withdrawal of the Hyper Studio-type software product from the market. One of the 
elementary schools planned to collect individual student data using Grady Profile on 
Newtons and then transfer files to larger classroom computers. Their technology 
specialist spent large amounts of time and many telephone hours with technical 
support trying to transfer files successfully. Due to such delays, magnet staff decided 
to pilot their electronic portfolios with samples of students during the first year of the 
grant, make revisions and implement them with larger student groups in year two, and 
complete portfolios across an entire grade level in year three. First year student 
samples are described in Table 1; Tables 2 and 3 show the evolution of these plans in 
the second and third years of the grant. 

Professional Development 

Professional development for grant staff and appropriate teachers at their 
schools began in year one. Magnet staff attended national alternative assessment and 
technology conferences and visited other schools and systems using technology to 
develop student information management systems and/or electronic portfolios. A 
consultant from the federal regional educational laboratory in Greensboro, NC 
(SouthEastem Regional Vision for Education, SERVE) provided a three-day workshop 



ERIC 



7 



Penta, page 4 



Table 2. Schools & Magnet Themes, Year 2 
Software, Student Sample, Professional Development, and Type of Assessment 
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Table 3. Schools & Magnet Themes, Year 3 
Software, Student Groups, Professional Development, and Type of Assessment 
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in alternative assessment, which included a train-the-trainers component so staff could 
begin to work with teachers at their schools. With electronic portfolios, it was also 
necessary to provide specialized technology training for school staff members. Staff 
development, in alternative assessment and in technology, continued in years two and 
three (Tables 2&3). 

Training in the structure and development of portfolios was more successful 
than technology training for the electronic aspects of the portfolios. Teachers, many of 
whom use some form of traditional portfolios for students, had a frame of reference for 
this aspect of the training. But most of them approached technology training with the 
misgivings and anxieties typical of adults facing some new requirement of the 
computer age. Glitches and user-unfriendliness of some of the portfolio software did 
nothing to allay that anxiety. 

The Pioneer Trail 

The difficulties in finding the time in teachers' already overcrowded schedules 
for training in both assessment development and software use were a challenge. Once 
time was found, the problems that even computer-sawy teachers had mastering the 
software elicited some "circle the wagons" feelings. Such sentiments occurred earlier 
for some of the technology specialists who logged innumerable hours trying to get the 
software to function properly so teacher training would go smoothly. Schools that 
used Hyper Studio had fewer problems because this was not a new product for them. 
But even at these schools, difficulties arose with regard to storage capacity for 
computer files as large as the portfolios. Hard drives, even on the new computers 
purchased through the grant, quickly filled up, leaving insufficient room for students to 
work on or store their portfolios. Staff members anticipated that the LANs would 
solve this problem once they were up and running at the three schools expecting them. 
The November 1995 date for this passed with assurances that the networks would be in 
place "after the holidays." As of April 1998, two of the three schools are still waiting. 
(No disparagement of the WCPSS technology department is intended; delays such as 
this are related to the enormity of installing networks in a system of over 100 schools 
serving almost 90,000 students, rather than any deficits in the department.) 

Not only were computer drives filled up, computer labs filled up also. Large 
numbers of students and teachers needed ready access to computers for structuring the 
portfolios and entering or scanning in work. Without easy assess to networks, schools 
had to rely on their existing computer labs. At most schools, these laboratories stay 
fully scheduled for technology classes as well as for the state-required computer 
competence testing. Booking blocks of lab time became quite a challenge; and, rather 
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than circling the wagons, staff members became very creative at this. For example, 
some teachers took their classes to the labs during homeroom periods. At other 
schools, electronic portfolio development was taught as an elective or combined with 
other electives that were already scheduled into the computer labs. 

The "circle the wagons" sentiment was strongest during a meeting of grant staff 
at the beginning of the 1997-98 school year — the third year of the three-year grant. 
Those attending were aware of the objective committing them to develop "new 
methods of assessing student progress." In spite of time and effort devoted to it thus 
far, the objective seemed beyond reach because of technology problems associated 
with the electronic portfolios. It seemed that the decision to develop electronic 
portfolios had been made just ahead of the time when such an undertaking would have 
been more feasible. For example, when completed, installation of LANs at every 
school will provide sufficient storage space and allow student and teacher access from 
classrooms as well as computer labs. The staffs review of software coming onto the 
market indicated that products are developing better portfolio capabilities and 
becoming easier to use. The development of writeable CD ROMs and wider 
availability of Jazz and Zip drives will help with storage problems. 

For the time being, it was clear to those meeting in September of 1997 that the 
major difficulties were with technology, rather than with the portfolios themselves. 
Only a certain portion of the technology problems could be remedied in the coming 
year. But the portfolios themselves, rather than their electronic aspects, could be 
emphasized for year three and could provide an effective means of fulfilling this grant 
objective. In the remainder of the September meeting and in subsequent meetings 
during the fall, staff members made plans to troubleshoot technology problems 
whenever possible but to emphasize the old-fashioned, paper-and-pencil aspects of the 
portfolios. Through this plan, some schools were able to develop full-fledged 
electronic portfolios for an entire grade-level of students; whereas others stressed 
traditional portfolios for all students, with small groups of students putting theirs into 
electronic format through previously scheduled computer electives. Whatever the plan 
(see Table 3), the portfolios themselves became an effective assessment method. And, 
because staff working on portfolio development for this grant did not circle the 
wagons, they learned lessons that will be valuable to others beginning a similar 
journey. 
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Lessons Learned 



Analyses of the various difficulties associated with alternative assessments are 
beginning to appear in the literature (Hardy, 1996; Monk, 1996; Stecher & Klein, 
1997). Results of this study are consonant with those findings, especially with regard 
to the incremental difficulties when new assessments are combined with technology to 
create electronic portfolios. 

Based on analyses of information from the multiple data sources in this study, 
several categories of difficulties and benefits have been identified. Difficulties had to 
do with aspects of technology such as: access to and/or purchase of computers; 
obtaining electronic portfolio software and training teachers and students to use it; 
ensuring access to portfolios while maintaining appropriate security; finding sufficient 
electronic storage space for multimedia segments of portfolios; and installing school- 
wide computer networks and servers to support portfolios. 

Benefits having to do with technology included improvements in teacher and 
student computer skills; storage of and access to paper-and-pencil student work as well 
as artwork, photos, and audio- and video clips; and the ability to integrate instruction 
when students and teachers can access portfolios in any subject from any classroom. 
Because LANs are still not fully functional at study schools, some potential benefits 
have not been realized yet: storage space for graphics-rich portfolios remains a 
problem and access to portfolios is still mainly through the computer labs rather than 
in classrooms (Table 4). 

This is a time when many schools and school systems are developing or 
planning to develop portfolio systems, and many of these will be coupled with 
technology to develop electronic portfolios. Such an approach is espoused in the 
literature, but there is a dearth of information from those who have already undertaken 
the journey. The lessons learned in this study are timely and should be of real 
assistance to others beginning a similar journey. 

1 . It may be better to develop effective traditional (i.e., not electronic) portfolios 
first and mediate them later. By doing both at the same time, schools in this study 
more than doubled the time and effort needed. And at times the technology 
overshadowed the curricular goals that the portfolios were designed to serve 
(Harrington-Lueker, 1997). The staff decision in September 1997 to re-focus on the 
traditional aspects of the portfolios brought curriculum rather than technology issues to 
the fore. Thus, the final portfolios, though not all in electronic format, were aligned to 
the curriculum and contributed to effective planning and implementation of 
instruction. 
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Table 4. Storage Methods Used in Year 3 



CD 




i n 




2. Results of this study, like Harrington-Lueker's, indicate that schools should 
"start small" with this type of technology. The original vision of the grant team was to 
develop electronic portfolios for all students in all grades within three years. As 
described earlier, technology delays quickly affected that timeline. Schools then 
worked with samples of students at a single grade level, eventually including entire 
grade levels by the end of the project. 

3. Take time to fmd the right software for electronic portfolios. Schools in this 
study tried four different software programs, none of which handled well both the 
database and multimedia aspects needed for electronic portfolios. Also invest the time 
to use software before adopting it; do not rely solely on producers' brochures and 
demonstrations. Pre-plan the portfolio format and contents and then pilot that with any 
software you anticipate purchasing. 

4. To develop and maintain electronic portfolios, both students and teachers need 
access to portfolios at anytime from anywhere in the school. This means that a local 
area network should be fully functional before undertaking a project of this sort. The 
LAN allows access, and it allows storage space. Electronic portfolios, particularly 
those including multimedia, require large amounts of storage. 

5. Technology support is essential. The staffing for this project was a single one- 
third to one-half time position for computer support at each school. All technology 
specialists in those positions were well-trained and experienced but could not provide 
sufficient coverage in part-time positions. They also had to contend with major 
software difficulties of programs not completely suited to the needs of electronic 
portfolios. When using software this complicated, it can be anticipated that 
technology staff will need time to maintain software, to troubleshoot, and to train and 
support teachers. 



Baron, J.B. , & Wolf, D.P., (Eds.). Performance-based student assessment: 
Challenges and possibilities . (The National Society for the Study of Education Ninety- 
fifth Yearbook). Chicago: The University of Chicago Press. 

Fogarty, R. (1996). Student portfolios: A collection of articles . Palatine, IL: IRI 
Skylight Training and Publishing, Inc. 

Greene, J.C., & Caracelli, V.J., Eds. (1997). Advances in mixed-method 
evaluation: The challenges and benefits of integrating diverse paradigms . New 
Directions for Evaluation, No. 74, American Evaluation Association. Jossey-Bass 
Publishers: San Francisco. 



References 



ERIC 




Penta, page 8 



Hardy, R. (1996). Performance assessment: Examining the costs. In Kane, 
M.B., & Mitchell, R. (Eds.), Implementing performance assessment: Promises, 
problems, and challenges (pp. 107-117). Mahwah, NJ: Lawrence Erlbaum Associates. 

Harrington-Lueker, D. (1997). Technology works best when it serves clear 
educational goals: Putting learning first is the key to using technology effectively. The 
Harvard Education Letter. XIII(6), 1-5. 

Herman, J.L., Aschbacher, P.R., and Winters, L. 1992). A practical guide to 
alternative assessment . Alexandria, VA: Association for Supervision and Curriculum 
Development. 

Kiger, D. M. (1997). Using staff values to develop assessment principles for 
high-stakes testing. ERS Spectrum: Journal of School Research and Information . 15(2), 
35-40. 

Messick, S. (1995). Studies of reliability and the validity of standards in 
performance assessment. Educational Measurement: Issues and Practice . 14(4), 5-8. 

Mitchell, R. (1992). Testing for learning: How new approaches to evaluation 
can improve American schools . New York: The Free Press. 

Monk, D.H. (1996). Conceptualizing the costs of large-scale pupil performance 
assessment. In Kane, M B., & Mitchell, R. (Eds.), Implementing performance 
assessment: Promises, problems, and challenges (pp. 119-137). Mahwah, NJ: 

Lawrence Erlbaum Associates. 

Shapley, K.S., & Pinto, M. F. (1997) Implementing portfolio assessment in the 
primary grades: teachers' perspectives. ERS Spectrum: Journal of School Research and 
Information . 15(2), 41-47. 

Stecher, B.M., & Klein, S.P., (1997), The cost of science performance 
assessment in large-scale testing programs. Educational Evaluation and Policy 
Analysis. 19, (1), 1-14. 

Wake County Public School System (WCPSS). (1995). Magnet Schools 
Assistance Program Approved Grant Application, U S. Department of Education, PR# 
S165A50147-95. 



U. S. Department of Education 

Office of Educational Research and Improvement (OERI) 

National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

rr TM030140 

REPRODUCTION RELEASE 



(Specific Document) 


AERA 


1. DOCUMENT IDENTIFICATION: 




Titlei-ZTs O ^cLl /’A*. LiJa^owS'? 

2zlSLC^lr0fv',<L- * 


Authors): /Vj^r-y 


Corporate Source: &)s /fe/ZA 


Publication Date: 



II. -REPRODUCTION RELEASE: 





In onJer to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources In Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Servioe (EDRS). Credit is given to the source of each document, and, if 
reproduction release is granted, one of the following notices is affixed to the document 



If permission is granted to reproduce and disseminate the identified document please CHECK ONE of the following three options and sign at the bottom 
of the page. 



The sample abetter shown below vrf!! be 
affixed to all Level 1 documents 


The sample sticker shown below wfl) be 
affixed to all Level 2A documents 


The sample sticker shown below wtt be 
affixed to all Level 2B documents 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE. AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY. 
HAS BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 








A® 




A® 






cj 




cjy 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


1 








2A 




2B 


1 


Level ‘ 

i 


1 

/ 


Level 2A 

t 


Level 2B 

t 




l/ 




□ 


□ 


Check hers for Level 1 release, permitting reproduction 
end dtsserrvnebon In rrecrofltfve or other ERIC wchtval 
media (e.g.. electronic) and papa r copy. 


Check hen for Level 2A release. permitting reproduction 
end dissemination m mtoofkt* and m electronic media 
ter ERJC archival oobecbon subscribers only 


Check here for Level 26 release, permitting 
••production end dlseeminebon In microfiche only 



Document* wB be proeetsed « Mctted prmMed reproduction quflty penriti. 

« permtuion to reproduce I* granted, but no box * checked, document* win be preoeaeed el Level 1. 



Sign 

here,-* 

please 



1 hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this document 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees end its system 
contractors requires pemission from the copyright holder. Exception is made for non-profit reproduction by libraries and other servioe agencies 
to satisfy mfomation needs of educators in response to discrete inquiries. 




Printed NemefPoetorVTsie: ; 

AWy <G -Pesrh, 




W-i rso-tHto 




E-Mail Address . 








(over) 



III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, If you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless It is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clear^^^^^Y QF MARYLAND 

ERIC CLEARINGHOUSE ON ASSESSMENT AND EVALUATION 
1129 SHRIVER LAB, CAMPUS DRIVE 
COLLEGE PARK, MD 20742-5701 
Attn: Acquisitions 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 

ERIC Processing and Reference Facility 
1100 West Street, 2 nd Floor 
Laurel, Maryland 20707-3598 

Telephone: 301-4974080 
Toll Free: 800-7994742 
FAX: 301-9534263 
e-mail: ericfac@ineted.gov 
WWW: http://ericfac.piccard.csc.com 

PPF-088 (Rev. 9/97) 

Yi- iVIOUS VERSIONS OF THIS FORM ARE OBSOLETE. 



